An Approximation-Based Data Structure for Similarity Search

Many similarity measures for multimedia retrieval, decision support, and data mining are based on underlying vector spaces of high dimensionality. Data-partitioning index methods for such spaces (e.g. grid-les, quad-trees, R-trees, X-trees, etc.) generally work well for low-dimensional spaces, but perform poorly as dimensionality increases|a phenomenon which has become known as thèdimensional curse'. In this paper, we rst provide an analysis of the nearest-neighbor search problem in high-dimensional vector spaces. Under the assumptions of uniformity and independence, we establish bounds on the average performance of three important classes of data-partitioning techniques. We then introduce the vector-approximation le (VA-File), a method which overcomes the diiculties of high dimensionality by following not the data-partitioning approach of conventional index methods, but rather a lter-based approach. A VA-File contains a compact, geometric approximation for each vector. By rst scanning these smaller approximations, only a small fraction of the vectors themselves must be visited. Thus, the VA-File acts as a simple lter, much as a signature le is a lter. Performance is evaluated on the basis of both synthetic and real data sets, and compared to that of the R ?-tree and the X-tree. We show that performance does not degrade, and even improves with increased dimensionality. Both our analytical and our experimental results suggest that the VA-File is generally the preferred method for similarity search over moderate and large data sets with dimensionality in excess of around ten.